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Abstract. The work is devoted to the formal verification of specifications over gen- 
eral discrete-time Markov processes, with an emphasis on infinite-horizon proper- 
ties. These properties, formulated in a modal logic known as PCTL, can be expressed 
through value functions over the state space. The main goal is to understand how 
structural features of the model (primarily the presence of absorbing sets) influence 
the uniqueness of the solutions of corresponding Bellman equations. Furthermore, 
this contribution shows that the investigation of these structural features leads to new 
computational techniques to calculate the specifications of interest: the emphasis is to 
derive approximation techniques with associated explicit convergence rates and formal 
error bounds. 



1. Introduction 

The use of formal verification notions and methods for dynamical systems has re- 
cently become an active area of research in systems and control theory [Tab09] . One 
of the most efficient techniques is model-checking, which aims at determining the sat- 
isfaction set of a given specification, i.e. the set of all states that initialize realizations 
verifying that specification. Probabilistic Computation Tree Logic (PCTL) is a modal 
logic which is widely used in formal verification and dependability analysis to express 
specifications for discrete-time probabilistic processes [BK08, Chapter 10]. The special 
case of discrete-time Markov Chains (dtMC) - models over discrete (countable) spaces 
- is well-studied in the literature and PCTL specifications can be verified over these 
models in an automatic manner by employing computationally advantageous proba- 
bilistic model checking techniques [HKNP06, KKZ05]. PCTL model checking has also 
been validated over numerous compelling applications [FKNP11]. 

The formal extension of PCTL to discrete-time Markov processes (dtMP) over gen- 
eral (uncountable) state spaces has only recently been discussed in [Hut05, RCSL10]. 
The latter work in particular has expressed the satisfaction set of a given PCTL specifi- 
cation as the level set of an associated state-dependent value function, and has further 
characterized the computation of such value function via dynamic programming (DP) 
[BS78]. Within PCTL, there is a clear distinction between finite-horizon specifications 
(the satisfiability of which depends on finite realizations of the system) and infinite- 
horizon specifications (those characterized over infinite paths) . In the context of dtMC 
with a finite state space, DP over a finite horizon is performed by iterative matrix mul- 
tiplications, whereas DP over an infinite horizon is reduced to solving systems of linear 
equations. On the other hand, over a general state space the corresponding proce- 
dures - namely Bellman iterations and Bellman equations - involve integral operators. 
Recent work (see e.g. [APLS08]) has shown that explicit analytical solutions over un- 
countable state-spaces are not to be found in general, and has stressed the need for 
methods to compute value functions with any given precision. 
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In the context of dtMP the work in [Hut05] has put forward finite abstractions 
[Pap03], where measures are approximated by monotone functions of sets. Although 
these abstractions are sound and upper and lower bounds for the expression of value 
functions have been derived [Hut05, Theorem 33] , no method to tune them has been 
given. Also, their tightness and usefulness or possible triviality (i.e. conditions for the 
errors to lie within (0,1)) has not been addressed. [RCSL10], in turn, has charac- 
terized PCTL specifications and their associated value functions with an emphasis on 
the issue of uniqueness of solutions of the related Bellman equations. The following 
questions have been left open to investigation: 

(1) how to compute finite-horizon value functions in PCTL with any given preci- 
sion? 

(2) since in general value functions are not known exactly and satisfaction sets are 
expressed as level set of these function, how to verify nested PCTL formulae 
(namely, specifications where the satisfaction set for the first formula appears 
in the definition of a second one)? 

(3) how to verify infinite-horizon PCTL specifications in PCTL, particularly if the 
sufficient conditions for the uniqueness of solutions of Bellman equations in 
[RCSL10] are not satisfied? 

With focus on 1), finite-horizon computations have recently received considerable 
attention. For discrete-time Stochastic Hybrid Systems (a class of dtMP), the work 
in [AKLP10] has put forward finite abstraction techniques to perform DP iterations 
over corresponding finite state-space dtMC. These results have been further sharpened 
in [SA11], where abstractions by state-space partitioning are obtained adaptively in 
accordance to a specification-dependent error. In both works the explicit abstraction 
error grows linearly with the time horizon of the corresponding PCTL specification, 
which does not allow applying the developed methods directly to the verification of 
infinite-horizon properties. 

This contribution of this work is hence focused on questions 2) and 3) and is 
twofold: the first goal is to complete the formal discussion on general state-space 
PCTL verification by dealing with nested formulae; the second goal (and the main task 
of this work) is to provide both analysis and computational tools for infinite-horizon 
PCTL specifications under conditions on the model that are as weak as possible and 
that are easy to verify. 

In order to address question 2), we introduce the concepts of sub- and super- 
satisfaction sets for PCTL specifications, the characterization of which requires only 
approximate knowledge of the corresponding value functions. Specifically, we show 
how the sub- and super-satisfaction sets of a nested sub-formula propagate to the cor- 
responding sets of the main formula: this is achieved by using monotonicity properties 
of corresponding value functions. 

In order to tackle question 3), we extend and generalize recent results in [TA11, 
TA12], showing that the sufficient condition provided in [RCSL10] for the uniqueness 
of the solution of a Bellman equation is only satisfied if the solution is trivial in some 
sense. We further show that a weaker version of this condition is both necessary and 
sufficient if the dtMP admits certain continuity properties. This result leads to novel 
techniques to solve Bellman equations whenever their solution is not unique, and pro- 
vides approximation techniques with associated explicit convergence rates and error 
bounds. These techniques are based on the reduction of the infinite-horizon problem 
to a finite-horizon one, for which computational methods available in the literature 
[AKLP10, SA11] can be directly applied. We furthermore discuss the relationship be- 
tween the issue of uniqueness of solution and the presence of absorbing sets over the 
(uncountable) state space: absorbing sets are shown to play a fundamental role for 
both the characterization and the computation of infinite-horizon properties. 
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The contribution is organized as follows. Section 2 introduces discrete-time Markov 
processes and PCTL specifications, and discusses the verification of nested PCTL for- 
mulae. Section 3 dives in depth into infinite-horizon problems. Section 4 provides two 
case studies to discuss the results and finally Section 5 concludes the work. 

Throughout the article we use tools of measure theory and of functional analysis. 
The following references can be consulted: [Dur04] for probability theory, [Rev84] for 
Markov processes and their kernel representation, and [Rud87] for functional analysis 
and measure theory. 

2. Markov processes and PCTL 

2.1. Discrete-time Markov processes. Let {X , 33) be a measurable space and let P : 
X x 33 — > [0, 1] be a stochastic kernel, so that P(-,B) is a non-negative measurable 
function for any set B e 33 and P(x, ■) is a probability measure on {X , 33) for any 
lef. The space of trajectories is denoted by n = ^" N ° (here N = N U {0}) and its 
product (T-algebra with & . It follows from [Rev84, Theorem 2.8] that there exists a 
unique discrete-time Markov process (dtMP) X = (X n ) n > with the transition kernel P, 
that is, for any x e X there exists a unique probability measure P x on {£!,&) such 
that P X (X Q = x) = 1, and for any measurable set B e S3 and any time epoch n > 

(2.1) P x (X n+1 e BIX,,,*!, . . . ,X n ) = P(X n+1 e B\X n ) = P(X n ,B). 

Equation (2.1) characterizes the Markov property and it indicates that the future of the 
process X n+1 is independent of its past history (X , . . . ,X n _ 1 ), given its current value 
X n . As a result, any dtMP can be characterized equivalently by the triple {X , 33, P). 

A familiar class of dtMP is that of stochastic dynamical systems. If (£„) n > is a 
sequence of iid random variables and / : X x R — > X is a measurable map, then 

(2.2) X n+l = f {X n ,E, n ), X = x^X, 

is always a Markov process characterized by a kernel Q(x,A) = v(£ e K : f{x, £) eA), 
where v is the distribution of £ . Conversely, any dtMPX admits a dynamical represen- 
tation as in (2.2), for an appropriate choice of the function/ [Kal02, Proposition 8.6]. 
However, theoretical studies of dtMP, as well as the current article, usually employ 
the representation via stochastic kernels. The reader interested in further discussions 
about modeling aspects of dtMP is referred to [Mey08, Appendix Al]. Among other 
models related to dtMP, Labeled Markov processes [DGJP04] may also be of interest. 

2.2. Probabilistic Computation Tree Logic (PCTL). PCTL is a modal logic employed 
to characterize classes of temporal properties of dtMC [BK08] and of dtMP [Hut05, 
RCSL10]. Properties are expressed as formulae in PCTL and are constructed accord- 
ing to the grammar of this logic. The grammar is based on AP, the set of atomic 
propositions, which can be thought of as tags or labels associated to the states of the 
model. Let AeAP and x e X; we write x |= A if the atomic proposition A is valid 
at state x. Since there is no substantial difference between A and its satisfaction set 
{x e X : x |= A} c X, we define atomic propositions to be measurable subsets of X, 
or equivalently AP c. 33, and require that f e AP. The grammar of PCTL is defined 
as follows. Atomic propositions are basic formulae that are used to build more com- 
plex formulae via logical rules. PCTL state formulae are subsets of X, whereas path 
formulae are subsets of f2. More precisely: 

• true is a formula with the whole X as its satisfaction set; 

• each atomic proposition A e AP is a formula with A itself as its satisfaction set; 

• if A and B are formulae, then so are -<A and AAB; 

• if 4> is a path formula and p e [0, 1], then P Mp [^] is a (state) formula, where 
ix can be any symbol from the collection {<,<,>,>}; 
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• if A and B are formulae and n e N , then XA, A U-" B, and A U B are path 
formulae. 

The semantics of PCTL state formulae is given as follows: 

x |= true for all xef 

x\=A <=> xeA 

x\=^A <=> x<EA € :=%\A 

x\=AAB <^> xeAnB 

With regards to path formulae, the meaning of XA (the next operator) is X 1 e A, thus 
x |= P Mp [XA] if and only if P(x,A) x p. The two additional path formulae depend on 
the bounded until operator U-" and on the unbounded until operator U . In order to 
characterize them through subsets of Q, let us introduce for any set A e 9& 

t a := inf{n > :X„ eA} 

to be the first hitting time of a set A over a realization X Q ,X 1 , Clearly, t a is a random 

variable with values in N U {oo}. We define A U-" B = {z B < t A c,t b < n} e 
whenever A,B e 3), which means that the path formula is satisfied over a trajectory 
for which B holds at least once within the n-step horizon, while A is persistently valid 
until that moment. Similarly, for the infinite-horizon case, we define 

All B = {T B < T A c, z B < oo}. 

To characterize satisfaction sets for until operators, we introduce the so called reach- 
avoid value functions: for any A,B e Sb, let us define 

w n (x;A,B) := P x (a U £ " b) , w(x;A,B) := P x (AUB), 

which leads to expressing [A U-" B] = {x e : w n (x;A,B) tx p}. Functions w n ,w 
are measurable, thus all PCTL formulae are well-defined measurable subsets of X and 
all path formulae are elements of & [RCSL10]. 1 

Let us provide a few examples: if A, B are PCTL formulae, then P< .os [A U P <: [XB] ] 
is a PCTL formula. Likewise, A => P> .9s[A U B] is a PCTL formula, since A => B := 
-tA V B and A V B := -i(nA A -.B). However, P >0 [(XA) A (B U C)] is not a PCTL for- 
mula, since the logical operation A is defined for state formulae but not over path 
formulae. Furthermore, PCTL path formula 0~"A := true U-" A= {t a < n} is known 
as a reachability event for a given set A and relates to a wide and important class of 
problems in systems and control [APLS08] . Its dual, the invariance (or safety) event 
□-"A = -i (<)-"A c ) = {t A c > n}, cannot be directly expressed in PCTL since the nega- 
tion of path formulae is not allowed. On the other hand, 

P x (t ac >n)=l-w n (x;^,A c ), 

thus one can define P Mp [□-"A] = P M '!_ p [true U-" A c ] , where the symbol <' stands 
for >, the symbol <' stands for >, and vice versa. We denote the invariance value 
functions by 

(2.3) u„Qc;A) := 1 - w n (x; 3C ,A C ), iz(x; A) := 1 - w(x; 5K",A C ). 

The results for reach-avoid and invariance given in this work can thus be directly ex- 
ported to the reachability property The latter represents also a crucial property for 
other types of logics, for instance linear temporal logic (LTL) [BK08, Chapter 5] . In 
particular, [AKM11] has argued that the verification of LTL specifications over a dtMP, 



1 Although the theory in [RCSL10] has been developed for models with X carrying a topological struc- 
ture, all the results on measurability hold without this requirement and as such they are also valid in the 
present instance. This work resort to a topological structure over the state space only in Section 3.2. 
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expressed via specific automata, can be reduced to a reachability problem [AKM11, 
Theorem 4] . 

2.3. Nested PCTL properties. As mentioned in the introduction, it is in general not 
expected that the value functions w n and w can be expressed explicitly. An alternative 
goal is the following [AKLP10] : given any precision level 5 > 0, find approximate 
functions w n and w such that |w n (x) — w„(x)| < 5 and |w(x) — w(x)| < 5, for all 
ief. Consider however the formula P> pi [a U P< P2 [B U C]l : if the value function 
w(x;B, C) can only be characterized approximately, what set should be considered to 
characterize P< P2 [B U C]? And how could this set be used in the parent formula? To 
resolve this issue we need the following fact. 

Proposition 1. Let AC. A* and B CB* be elements of 3& and let n e N . For all x e 3C: 
w n (x;A,B) < w„(x;A*,B*), w(x;A,B) < w(x;A*,B*). 

Proof. Since {t b < t A c, t b < n} c {t b « < T^y, t b , < n] the proof immediately follows 
as the probability measure P x is a monotonic function of sets for any xef. □ 

For a PCTL formula A e 38, we say that A* (A*) is a subsatisfaction {supersatisfaction) 
set if A* c A (AC A*). Clearly, A„ denotes a conservative set, the states of which also 
satisfy A, while A* denotes a relaxed set: any state in (A*) c does not satisfy A either. 

As done above, let w n ,w denote some abstract 5 -approximations of w n and w, re- 
spectively. Let us show as an example, how the formula P> pi [a U P< P2 [B U C]] can 
be verified. Since 

w(x;B,C)-5 <w(x;B,C)<w(x,B,C) + 5, 

it follows that w(x;B,C) <p 2 — 5 implies w(x;B, C) < p 2 , and that w(x;B, C) > p 2 + 5 
implies w(x;B, C) > p 2 . As a result, if we denote D = P< P2 [B U C], then the sets 

D* = {x e X : w(x;B, C) < p 2 - 5], D* = {x e X : w(x;B, C)>p 2 + 5} 

represent sub- and super-satisfaction sets for D. Finally, from Proposition 1 we obtain: 

£„ = (xef : w{x;A,D lf )>p 1 + 5}, £ ! = {ief :w(x;A,D*) < p 1 - 5} 

are sub- and super-satisfaction sets for P> Pi [a U P< P2 [B U C]] . The application of this 
procedure over formulae including the operator X is direct, since P(x, •) is a monotonic 
function of a set-valued argument for any x 

A general algorithm for the verification of nested formulae follows: given the ability 
to approximately compute value functions with a precision 5, find sub- and super- 
satisfaction sets for the sub-formulas on the lowest level (leaves) of a given formula 
tree, then use these sets to find sub- and super-satisfaction sets for higher-level for- 
mulae inductively, until the sub- and super-satisfaction sets for the given formula are 
found (at the root). 

3. Verification of infinite-horizon PCTL specifications 

The goal of this section is to investigate the verification of infinite-horizon PCTL 
specifications and to provide methods to compute associated value functions with any 
given precision. For this purpose Section 3.1 introduces DP techniques to characterize 
the corresponding value functions, points out related issues in their evaluation and 
provides sufficient conditions for the precise reduction of infinite-horizon problems to 
finite-horizon ones. In Section 3.2, the concept of absorbing set is used to show that 
for a class of problems the above conditions are also necessary, and that they relate 
to the uniqueness of the solution of Bellman equations. This result is further applied 
to derive methods to solve Bellman equations with non-unique solutions, both in the 
general case (which is done leveraging Lyapunov-like locally excessive functions - cfr. 
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Section 3.3), and in the special case of integral kernels (where such functions are not 
needed - cfr. Section 3.4). The presented techniques depend on the characterization 
of absorbing sets, which is discussed in Section 3.5. 

3.1. Dynamic programming and Bellman equations. Let B denote the space of all 
real-valued, bounded and measurable functions, which is a Banach space with a norm 
given by ||/|| := sup|/(x)| for / e B. An operator £ : B — » B is called linear if 

J (af + I3g) = a^{J) + fij(g) for any a.jjel and /, g e B. The quantity 

(3-D 11/11 := sup ||// 1| 

ll/ll<i 

is called the norm of the linear operator Jf. We say that J? is a contraction if < 
p < 1. An important example of a linear operator associated to a dtMP is the transition 
operator g? : B — » B given for any function / e B by the following formula 



r 



f{y)P{x,Ay). 



Let us furthermore introduce an invariance operator J A , parameterized by a measur- 
able set A e S3, and given by J?jJ(x) = l A (x)2? f(x). Clearly, J A is also a linear 
operator and J x = & . Moreover, J A is a monotone operator, which means that for 
all functions /, g e B and any set A e 53 it holds that J^/ (x) < J A g{x) for all x e 
whenever /(x) < g(x) for all x e $f . As an abbreviation, for a function g : .Sff — » R 
and a constant 5 e K we use {g < 5} = {x e : g(x) < 5}; a similar notation is used 
for any of the other symbols in the collection {<,>,>,=}. 

Let us introduce a DP procedure for until-like specifications in PCTL. Let A, B e 9& 
be given sets (equivalently state formulae in PCTL). From [RCSL10, SL10] it follows: 



(3.2) 



aientiy state tormuiae m j^lilj. from [KL^lj 

iw n+l {x;A,B) = l B (x) + J? A \ B w n (x;A,B), 
\w (x;A,B) =l B (x). 



The computation in (3.2) involves iterations of the integral operator J?a\b- Results in 
[AKLP10, SA11, SA12] allow one to compute a piece-wise constant function approx- 
imation w n , which is such that \\w n — w n \\ < An, where the constant X depends on 
the quality of the state space partitioning (see e.g. [SA11, Theorem 4]). Thus, in the 
remainder of this work we assume that finite-horizon problems can be solved approx- 
imately and with any given precision by any of the techniques given in the literature, 
and instead focus on the reduction of infinite-horizon problems to finite-horizon ones. 
For infinite-horizon problems, it holds that w(x;A,B) = lim w n (x;A,B), where the 

n— »oo 

limit is point -wise non-decreasing [RCSL10]. In [RCSL10, Lemma 5] the monotone 
convergence theorem is applied to w n — » w, in order to show that the function w solves 
the fixpoint Bellman equation 

(3.3) w{x;A,B) = l B (x) + J? AB w(x;A,B). 

However the convergence of w n — » w is not necessarily uniform. Moreover, equation 
(3.3) may have multiple solutions: since it is an affine equation, if it does not have a 
unique solution then it admits infinitely many, spanning an affine subspace of B. To 
further look into this issue we leverage value functions for invariance. As discussed 
above, the until specification can be used to express the invariance over a given set 
A e 83. Using formulae (2.3) and (3.2) we obtain the following DP recursion 



(3.4) 



u n+1 (x;A) = J A u n {x;A), 
u [x;A) = l A (x). 
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It easily follows that u n converges point-wise non-increasingly to function iz, thus 

(3.5) u(x;A) = J A u{x;A). 

Clearly the verification of the invariance specification inherits issues of non-uniform 
convergence and of non-uniqueness of the Bellman equation (3.5) from the until spec- 
ification in (3.3). However, the Bellman equation for the invariance specification has 
the advantage of being linear and thus always admits the trivial solution iz = 0. More- 
over, the analysis of the affine equation on a linear space can be reduced to the analysis 
of its homogeneous (linear) version: dealing with (3.5) leads to finding methods for 
solving (3.3) as well. 

Remark 1. There exists a least fixed-point characterization for the infinite-horizon value 
functions [RCSL10, Lemma 6]: w(x;A,B) is the least non-negative solution of (3.3), 
i.e. if f is any other non-negative solution of (3.3), then w(x;A,B) < /(x) for all 
x e X '. As a result, u(x;A) is the largest solution of (3.5) not exceeding 1. Although such 
characterization adds little to the computation ofu and w, it results in the useful fact that 
\\u\\ = 1 whenever u is non-trivial, namely whenever u is not identically equal to zero. 

A sufficient condition for the uniqueness of the solution of (3.5) is that ||u 1 (-, J 4)|| < 1 
[RCSL10, Proposition 7], which in turn leads to the contractivity of the operator J A . 
While this condition may be easy to check, it can be restrictive: in this case (3.5) 
admits the unique solution u = 0. As a result, any invariance problem with a non- 
trivial solution will not satisfy this sufficient condition. It follows that the weaker 
condition ||u„(-,A)|| < 1, for some n > 1, is also sufficient for the uniqueness of the 
solution of (3.5). Let us introduce the quantities 

m(A) = inf {m > : ||u m (-,A)|| < 1} , p(A) = ||u m(A) (-,A)|| , 

for any A e 08, where we set p(A) := 1 if m(A) = oo. The quantity m(A) is discussed in 
more detail for the special case of Markov Chains in Section 3.4. 

Proposition 2. Let Ae 33 and denote for simplicity m := m(A) and p := p(A). Then: 

i. ifm< oo, then u(-,A) = 0, andfor alln>0 it holds that ||u n (-;A)|| < p LmJ; 
it ifA,B e 03 are disjoint 2 and m < oo, then for alln>0 

m I „ I 

(3.6) 0<w(x;A,B)-w n (x;A,B)< - pL^J. 

1-p 

Proof. For part (i), we have from (3.4) that u n = {J A ) n ~ k u k , for all < k < n. Clearly, 
from the finiteness of m and the definition of p it follows that u m (-;A) < l A (-)p, so 

u n (-,A) < p • (j^)"- m l A (0 = pu n _ m {-,A). 

for n > m. By induction we obtain that ||u„(-;A)|| < p L™J 3 so that 

u(-,A) = lim u„(-;A) = 0. 

n— »oo 

For part (ii), we define functions A n (x) := w n+1 (x;A,B) — w„(x;A,B). Clearly, 
it holds that A (x) = l A (x)P(x,B) and A„ +1 (x) = J^ A A„(x). Moreover, from the 
fact that A (x) < u (x;A) and the monotonicity of the operator J A , we have that 
A n (x) < u„(x;A). It further follows that 

oo oo oo m 

w(x;AB)-w n (x;A,B) = Y i A i (x)<Y i pl^ < ^ mp * = __ p LiJ. 

i=n i=n Jc=[n/mJ ^ 

□ 

2 In the following, for the sake of the simplicity the set-valued arguments of the reach-avoid value 
functions are assumed to be disjoint. This assumption does not affect the generality of the results, since 
w(x;A,B) = w(x;A\B,B) and hence any reach-avoid problem can be always considered as a problem on 
disjoint sets. 
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As mentioned before, one goal of this section is to reduce a given infinite-horizon 
problem to a finite-horizon one, with the ability to tune the error incurred in this 
reduction. If m(A) < oo, and since the right-hand side in (3.6) decreases exponentially 
fast with respect to n, Proposition 2 provides a method to achieve this. In the following, 
the condition m(-) < oo, for an appropriate set-valued argument, indicates that the 
corresponding infinite-horizon problem can be reduced (and thus solved). 

It is worth mentioning that Proposition 2 elucidates the difficulty in the direct exten- 
sion of the error bounds in [AKLP10, SA11] from finite- to infinite-horizon problems: 
the developed finite-horizon approximation techniques can be interpreted as providing 
a perturbation P of the original stochastic kernel P. Thus, they are tailored at render- 
ing the one-step error \\P — P\\ (under the operator norm in (3.1)) as small as possible. 
However, in general a bound on the one-step error cannot be extended over an infinite 
time horizon, as the following argument shows. Let us consider the case where the 
solution of the invariance problem on a set A for the dtMP {% ', 93, P) is non-trivial. 
We denote the corresponding value function as u(x;A). It follows from Remark 1 that 
1 1 u 1 1 = 1. Let v be any probability measure on (3£ , 93) such that v(A c ) > 0, and define 
P 5 [x;A) := (1 - 5)p[x;A) + 5v{A), for 5 e (0, 1). We have 



P 5 f-Pf\\ 



f(yMdy)-5-pf 



< 5 (11/11 + IIP/ID < 25| 



for any function /eB. Hence \\P 5 — P\\ < 25, so that it can be made arbitrarily small. 
On the other hand, if we denote by u 5 the solution of the invariance problem on A for 
dtMP {%, 93, P 5 ), we obtain that ||u^|| < 1 - 5 < 1. As a result, iz 5 = by Proposition 
2, so that ||iz — iz 5 || = 1, regardless of how small 5 is. 

3.2. Absorbing and simple sets. From Proposition 2 it follows that the condition 
m(A) < oo in particular implies the uniqueness of the solution of the corresponding 
Bellman equation. It turns out that under some continuity assumptions on the kernel 
P this condition is also necessary. Before we proceed, we introduce the notion of 
absorbing set, which is crucial for further discussions. 

Definition 1. A set A & S3 is called absorbing ifP{x,A) = 1, for all x e A. If for A e S3 
there is an absorbing subset A' C A such that A" C A' whenever A" C A is absorbing, 
then we say that A' is the largest absorbing subset of A and write A' = l.a.s.(A). The set 
A is called simple if it does not have non-empty absorbing subsets, i.e. l.a.s.(A) = 0, and 
non-simple otherwise. 

Clearly, the whole state space X and the empty set are always absorbing, and 
if (A„) n > is a sequence of non-empty absorbing sets, then their union A = [jA n is 

n 

absorbing and non-empty. However, it is by no means clear that l.a.s.(A) exists for any 
given set A, since A may contain uncountably many absorbing subsets and their union 
may not be even measurable. Surprisingly, invariance value functions are useful to 
show that l.a.s.(A) is always well-defined. 

Lemma 1. Let A e 93 and denote A n = {u n [-;A) =1} for all n > 0, so that A = A 

CO 

Further, letA^ = f) A n e S3, then for alln>0 it holds thatA n+1 C.A n and 

71=0 

(3.7) A n+1 = {xeA:P(x,A n )=l}. 

The set A^ admits the representation A m = {u(-;A) = 1} = l.a.s.(A), i.e. it is the largest 
absorbing subset of A In particular, if m{A) < oo then A is simple. 
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Proof. Let us first prove (3.7): for an arbitrary x eA„ +1 it holds that 
(3.8) 



r 



u n {y;A)P{x,Ay)- 



u n (y;A)P<ix,dy). 



P(x,dy) < 1 = u n+1 (x;A) = l A (x) 

A X A 

Thus f (1 — ii„(y;A))P(x,dy) = as it is non-positive from (3.8) and the integrand is 



A 



non-negative. Due to the latter fact, P(x, {u n (-;A) = 1}) = 1 or equivalently P(x,A„) = 
1. Conversely, xeAbe any state such that P(x,A n ) = 1 and let us show that x eA„ +1 . 
Indeed, 



r 



U;A) : 



u n {y; A)P(x, dy) > u n (y; A)P(x, dy) = P{x,A n ) = 1, 



thus x e A n+1 . Since A n+l = {ieA: P(x,A n ) = 1} and A x c A , then A 2 QA X , and 
by induction A n+1 QA n for all n > 0. If iz(x;A) = 1 for some xeA, then u„(x;A) = 1 
and x e A n for all n > 0, hence x e A M . If x e A^, then x e A n for all n > 0, so that 
iz(x;A) = lim u n [x;A) = 1. 

n— *co 

Suppose now that A is non-simple and that A' is its arbitrary absorbing subset. 
Clearly u(x;A) = 1 for all x e A', hence A' c A x . Furthermore, if A m ^ 0, then 
for any x e A M and n > it holds that x eA„ +1 , hence P(x,A„) = 1. This implies that 
A^ is absorbing since 

f °° \ 

P(x,A M ) = P x, f] A n = lim P(x,A„) = 1, 

V n=0 y 

which leads to conclude that A m is the largest absorbing subset of A. □ 

As it has been mentioned above, some continuity assumption on the kernel P are 
needed in order to sharpen the results. To do so, the state space needs to be endowed 
with a certain topological structure (see e.g. [HLL96]). 

Definition 2. A state space {% , !M) is called topological if X is a Borel subset of a Polish 
(i.e. a metrizable, complete, and separable) space and if ^ is a Borel a-algebra of X. A 
kernel P on a topological space is called Feller or weakly continuous if the function g? f is 
upper semi-continuous (u.s.c.) whenever f eB is u.s.c. [HLL96, Appendix C]. 

A dtMP [X , 3&, P) is said to be weakly continuous whenever (X , is a topological 
state space and P is weakly continuous. 

The next theorem shows that for a weakly continuous dtMR the infinite-horizon 
problem over a compact set A can be directly reduced to the finite-horizon one (in the 
sense that m(A) < 1) if and only if the set A is simple. 

Theorem 1. Let (X be a topological state space and A be a compact set. If Pis weakly 
continuous then l.a.s.(A) is a compact set and the following statements are equivalent: 

(1 ) m(A) < oo; 

(2) J?£ is a contraction on B for some finite n (contractivity); 

(3) equation (3.5) has a unique solution (uniqueness); 

(4) u(x;A) = 0/or allx&X (triviality); 

(5) the set A is simple: A m = (simplicity). 

Proof 1) ^> 2) Clearly, for any function / e B it follows that (x) < ||/||l A (x) for 
all states x &X. Thus if m(A) < oo, then j? m W +1 is a contraction since 

K m(A)+1 /ll < ll/ll • K mCA) iJ = 11/11 • IKca)G;a)|| < p(a)||/||. 
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2) => 3) If / G B be a solution of (3.5), i.e. / = JjJ . By induction we have 
/ = «^a/> which by contraction mapping theorem [Rud76] implies the uniqueness of 
the fixpoint / . 

3) => 4) follows from the linearity of (3.5) and 4) => 5) from Lemma 1, so we only 
have to show that 5) => 1). Suppose this is not true, i.e. m(A) = oo but A is simple. It 
follows that A n / for all n > 0. Since A is compact and X is metrizable, A is closed 
and hence u = 1 A is u.s.c Hence u n is u.s.c for all n > by the weak continuity 
of P, which implies that all sets A n = {u„(-;A) > 1} are compact. Moreover, they are 
not empty and so their intersection A^ is compact and non-empty which leads to a 
contradiction. □ 

Remark 2. Within the main goal of reducing infinite-horizon problems over a set A to 
a finite-horizon ones, let us remark that numerical methods for finite-horizon problems 
leading to the computation of PCTL value functions with any given precision have been 
developed, up to our knowledge, only for compact subsets of finite- dimensional metric 
spaces [AKLP10, SA11] - this aligns with the assumption raised for Theorem 1. Also, 
there conditions required on the kernel P are stronger than the weak continuity raised 
above. Taking all of this into account, the assumptions in Theorem 1 are rather mild. 
Furthermore, some of the relations in the theorem are true under even weaker conditions: 
we postpone the discussion of these facts to Section 6 (Appendix). 

Remark 3. It follows directly from Theorem 1 that if m(A) < oo then j?™^* 1 is a 
contraction and furthermore, H^" || < p(A). 

Corollary 1 (From Theorem 1). Let (X , 33) be a topological state space and A be a 
compact set. J/u n (-;A) are u.s.c. functions in the subspace topology of A [Rud87] for all 
n G N , then l.a.s. (A) is compact and statements 1) - 5) in Theorem 1 are equivalent. 

Proof. According to the proof of Theorem 1, the implications 1) => 2) => 3) => 4) do 
not require weak continuity of P and thus hold in the current setting as well. Moreover, 
because functions u n are u.s.c, then 5) => 1) follows directly. □ 

3.3. A decomposition technique. Although Theorem 1 has been stated in terms of 
value functions for the invariance problem, its application to the issue of uniqueness of 
the solution of a reach-avoid problem is direct, since (3.5) is a homogeneous version 
of (3.3). As a result, if the dtMP (X , 33, P~) is weakly continuous, sets A,B are disjoint 
and A is compact and simple, then m(A) < oo and the reach-avoid problem can be 
solved. Thus, the next goal is to study the case of a non-simple set A. For this objective 
the characterization given in Theorem 1 is again useful. We proceed assuming that the 
l.a.s. of a given set is known, and leave the discussion on the characterization of the 
l.a.s. of a given set and the verification of the simplicity of a set to Section 3.5. 

If A is non-simple, the main issue preventing an efficient solution of the problem 
is the presence of an absorbing subset l.a.s. (A). This leads to the lack of contractivity 
of the operator J A and to the non-uniqueness of the solution of (3.3). Intuitively, 
if we were to remove some neighborhood C D l.a.s. (A), then we would expect that 
m(A \ C) < oo, so that a related problem can be solved on A \ C. Moreover, recall 
that the solution of the original problem on l.a.s. (A) is known: w(x;A,B) = for all 
x g l.a.s. (A), since such states initialize trajectories that never reach the set B. The 
following result relates the solutions of the two problems: 

Lemma 2 (Decomposition technique). Let sets A, B G 33 be disjoint, and set C g 33, C c 
A be such that the invariance value function u(-;A\ C) = 0. Then w(x;A\ C,B) is the 
unique solution of the following Bellman equation 



(3.9) 



w(x; A \ C,B) = l B (x) + J? Ac w[x;A \ C,B), 
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and for all x &X the following holds: 

(3.10) < w(x;A,B)-w(x;A\ C,B) < supw(y;A,B). 

Proof. As an abbreviation, let us denote t 1 = t BuC and t 2 = T (AuB y and let us partition 
the event space H by the following four disjoint hypotheses: 

H! = {t 2 < z 1 ,t 2 <oo}, H 2 = {t 1 =oo,t 2 = oo}, 

H 3 = { T l < T 2> T l < 00, Tj = T B }, H 4 = {Tj < T 2 , Tj < 00, Tj = T c }. 

Recall that w(x;A,B) = P x {t b < t 2 , t b < oo}, thus 

4 

w(x;A,B) = ^P x ({t b < t 2 ,t b < ooJnHj) . 
i=i 

Note that the first term is zero since clearly {t b < t 2 ,t b < 00} nH 1 = 0. The second 
term vanishes because P X (H 2 ) = u[x;A\C) = 0. Since H 3 c. {z B < t 2 ,t b < 00} the 
third term equals to P X (H 3 ) = w(x;A\C,B), which leaves only the fourth term to be 
studied. Let x be any such that P X (H 4 ) 7^ and define a measure v x on {X , 8$) by 

v x (D)=P x (X Tc eD|H 4 ), 

so that clearly v x (C c ) = 0. For such fixed x it holds that 

< P x ({t b < t 2 , t b < 00} n H 4 ) = P x ( {t b < t 2 , t b < oo}| H 4 ) P X (H 4 ) 



r 



= w{x;A,C)- 



w{y;A,B)v x {dy) < supw(x;A,B). 



The same bounds clearly hold in the alternative case P X (H 4 ) = 0. 

Finally, it follows that w{x;A \ C,B) is the unique solution of the corresponding 
Bellman equation (3.9) from C) = (see Proposition 5 in Section 6). □ 

Corollary 2. [From Lemma 2] Let {X , £%,P) be a weakly continuous dtMP and letA,B e 
^ be disjoint and such that A is a compact, non-simple set. Let C C A be an open 
neighborhood of\.a.s.(A) in the subspace topology of A. Then (3.10) holds for all x e 90 , 
and m(A \ C) < 00. 

Proof. Since C is open in A, the set A \ C is a closed subset of a compact set A and thus 
itself compact. From the inclusions l.a.s.G4) c C c A it follows that A \ C is simple, 
hence Theorem 1 ensures that all the conditions of Lemma 2 are satisfied. □ 

In order to render the result in Corollary 2 useful for the computation of the infinite- 
horizon reach-avoid value function, we should provide a method to choose an open 
neighborhood C of l.a.s.(A), such that supw[y;A,B) < e, where e > is a given 

precision level. We use the theory of excessive functions [SRG08] to achieve this goal. 

Definition 3. Given a function gel, the excessive set of g is S g = \3? g — g < 0}. If 
S g = X , i.e. if ' S? g{x)< g{x)for all x e X, then the function g is called excessive. 

The relation between excessive functions and infinite-horizon invariance is given 
via Doob's inequality [SRG08] : if g e B is an excessive, non-negative function, then 

n>0 J 

for all 5 > 0. The inequality (3.11) can be rewritten via the invariance value function: 
(3.12) u(x;{g<5})>!-^. 



(3.11) P x \snpg(X n )>5}< 
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Excessive functions for stochastic systems are akin to Lyapunov functions for deter- 
ministic systems, since they are characterized by decreasing behavior along the dynam- 
ics of the process, as the inequality g?g < g suggests 3 . As is the case with Lyapunov 
functions for deterministic systems, it is non trivial to find excessive functions. How- 
ever, it is possible to relax the assumption on global excessivity and to employ a local 
version of Doob's inequality. 

Lemma 3. [Kus67, Theorem 12] Let g e B be a non-negative function such that for 
some 5 > it holds that {g < 5} C g g . Whenever x e {g < 5}, it follows that 

P x (supg(Xj>5}<^. 

[ n>0 J O 

The idea behind the proof of this lemma is to consider a set A = {g < 5}. The 
related invariance value function does not depend on P(x, •) for x e A c , where it is 
simply equal to zero (recall that all the integrals in the DP recursion (3.4) are equiva- 
lently taken over the set A) . As a result, exclusively the dynamics within the set A are 
important for the process. 

Definition 4. For a topological state space {X , $)we say that a non-negative continuous 
function g e B is 5-locally excessive on the set A e Z% if for some real number 5 > it 
holds that {g = 0} = l.a.s.(A) and that {g <5}£A£ g g . 

Theorem 2. Let {X , £%,P) be a weakly continuous dtMP and let A,B e 9B be disjoint 
and such that A is a compact, non-simple set. If there exists a 5-locally excessive function 
gonA, then for any e e (0, 1) it holds that m(A \{g < e5}) < oo and that 

(3.13) 0<w(x;A,B)-w(x;A\{g <e5},B) <e. 

Proof. First, we show that for any e e (0, 1), if g(x) < e5 then w(x;A,B) < e. Indeed, 

g(x) 

as {g < 5} c g by Lemma 3 we have that iz(x; {g < 5}) > 1 — for all x e X , so 

5 

u(x;A) > 1 — 

o 

for all x e X , which follows from {g < 5} c A. Since u(x;A) = 1 - w (x; X, A c ), then 

w(x;X,A c )<^ 
o 

for all x e X, and since A c. X and B c a°, from Proposition 1 it follows that 
w(x;A,B) < As a result, for any x e {g < e5} it holds that w(x;A,B) < e. 

Second, let us fix any e e (0, 1) and denote C = {g < e5}. Clearly, l.a.s.(A) c c and 
the set {g < e5} is open in A since g is continuous on A. The statement of the theorem 
then follows from Corollary 2. □ 

3.4. Integral kernels and discrete-space Markov Chains. From Theorem 2 it follows 
that for weakly continuous dtMPs a reach-avoid problem on a non-simple set can be 
solved if an appropriate locally excessive function is found. For a known and studied 
subclass of these processes the problem can be solved even without resorting to such 
functions. We write P(x, dy) = p(x,y)/i(dy) if P is an integral kernel with a basis ju 



3 From the definition of 3" is follows that 2?g(x) = E x [g(Xj)], where E^. denotes the expectation with 
respect to P x . Thus the condition 2? g < g means that the expected value of the function g at the next time 
step is bounded by its current value, so that the function g does not increase on average along realizations of 
the dtME Thus the function g e B is excessive if and only if the process (g{X n )) n>0 is a P x -supermartingale, 
forallxs*" [PS06,p.20]. 
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and a density p, namely when /i is a a -finite non-negative measure, the function p is 
<% <g> ^-measurable 4 , and for any A e 9B it holds that 



P(x,A) : 



p(x,y)ju(dy). 



Furthermore, we raise the following assumption, which generalizes one used for re- 
lated studies over the finite horizon [AKLP10, SA11]. 

Assumption 1. Let {SC , SB) be a topological state space and let dbe a complete metric on 
SC consistent with the given topology. Assume that /i(A) < oo for any compact set A and 
that there exists a SB -measurable function A A , such that j A A A (y)/i(dy) < oo, and such 
that for all x',x" eAand ix-a.a. y & A 



(3.14) 



lp(x", y) -p(x' 3 y)\ < X A {y) ■ d{x',x"). 



Let us mention that if P(x,dy) = p(x,y)ju(dy), set C is such that ju(C) = 0, and set 
A is an absorbing set, then clearly A\ C is absorbing as well. This, in turn, implies that 
if A is a simple set then AU C is also simple whenever |U(C) = 0. 

For a topological space SC, let us use the following notation: let dB and B° be the 
boundary and the interior of set B c SC. 

Theorem 3. Let {SC , 9B~) be a topological space and P(x, dy) = p(x,y)ji((dy). Let sets 
A,B e 31 be disjoint and let the set Abe compact. Suppose that at least one of the following 
two items holds: 

i. P is weakly continuous; 

ii. Assumption 1. 

Whenever /x(3(l.a.s.(A))) = 0, it holds that m(A\ (l.a.s.(A))°) < oo and for all x e SC 
(3.15) w(x;A,B) = w(x;A\ (l.a.s.(A))°,B). 

Proof. To prove (3.15) we apply the decomposition technique in Lemma 2 over a set 
C := (l.a.s.(A))°. The result follows since w{y;A,B) = for all y e (l.a.s.(A))°, so we 
are only left to prove that for the set A := A\ (l.a.s.(A))° it holds that m(A) < oo. 
If i. is true, then l.a.s.(A) is compact by Theorem 1 so 

l.a.s.(A) = (La.s.(A))° u 3(l.a.s.(A)). 

As a result, the set A := A \ (l.a.s.(A))° is compact and simple since |i((3(l.a.s.(A))) = 0. 
From Theorem 1 it follows that m(A) < oo. 

If instead ii. is true, then u„(-;A) is a continuous function on A for any n > 0, since 



\u n+1 {x";A)-u n+1 {x';A)\ 



< 



f 



< 



u n (y;A) (p (x",y) -p (x',y)) M (dy) 
|p (x",y)-p(x',y)\rtdy) 



d{x',x"), 



J 



4 Here & ® S is the a-algebra on the set X x X generated by the class of the measurable rectangles 

33 x 33 = {B 1 xB 2 : B 1 ,B 2 e 33}. 
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for any pair of states x',x" e A. From Corollary 1 it follows that l.a.s.(A) is compact. 
Following the same argument as in case i., we obtain that the set A is compact and 
simple. Similar to (3.16) we obtain that u n (-;A) is a continuous function on A for any 
n > 0, hence m(A) < oo by Corollary 1. □ 

Remark 4. In the special case of the invariance problem over the set A, under the as- 
sumptions of Theorem 3 it follows that u(x;A) = w(x;A, (l.a.s.(A))°). Moreover, the 
proof of Lemma 2 implies that for any initial state xeJ, P x -a.s. a trajectory (X n ) n > 
of the dtMP that stays invariant in the set A necessarily reaches its largest absorbing sub- 
set. Altogether, this enlightens yet another interesting relation between invariance and 
reach-avoid problems. 

In case /x(d(l.a.s.(A))) > 0, Theorem 3 cannot be applied and Theorem 2 is left 
as an alternative to tackle this problem by coming up with an appropriate 5-locally 
excessive function for the given set A. In general such a function may not exist even 
if P is a weakly continuous dtMP and A is compact (see example in Section 4.1). 
However, if Assumption 1 holds it follows that the function u(-;A) is continuous on A 
- the proof follows the same lines as in (3.16) - and g(x) = 1 — iz(x;A) is a 1-locally 
excessive function on A. This consideration suggests that at least under Assumption 1 
there always exists a 5-locally excessive function over a compact set A. 

Let us now tailor the above results to the case where the state space is countable, 
i.e. when the process is a discrete-time Markov Chain (dtMC). The methods developed 
in this section are directly applicable to the dtMC framework where, to the best of our 
knowledge, available techniques allow one to compute infinite-horizon reach-avoid 
value functions as a limit of converging iterations, but without bounds on the error 
[BK08, Theorem 10.15]. 

Without loss of generality, let us assume that X — N is the state space of a dtMC. 
We endow X with the discrete metric d{i, j) := lyjCO, so that S3 = 2 X . The basis 
cr-finite measure is chosen to be the counting one: = 1, for any i e X . Any 
stochastic kernel P over (X , 0$) can be expressed as a matrix P = (p,j)y sN , where 
p i; - := P(i, {;}). With the chosen counting measure, the entries of the stochastic matrix 
P determine the density function, namely p{i,j) = py. Clearly, compact subsets of X 
correspond to finite sets, so they are of finite measure fx. Moreover, P is always weakly 
continuous, since on a discrete topological space every function is continuous. 

Remark 5. For a dtMC the largest absorbing subset of any finite set can be found algo- 
rithmically. Indeed, from Lemma 1 it follows that l.a.s.(A) = {u(-;A) = 1}, so the set can 
be equivalently expressed via a CTL formula: l.a.s.(A) = {x e A : x \= VDA}. As such, it 
can be computed in ^(/x 2 (A)) time over the dtMC graph (to be defined below) [BK08, 
Theorem 6.30]. 

Corollary 3 (from Theorem 3). Let A,B e 03 be disjoint and A be finite, denote A := 
A\l.a.s.(A) and b ; :=P(i,B) = XiPij- The reach-avoid value function w(i; A, B) is defined 

j€B 

uniquely by 



(3.16) 



w(i;A,B)=l ifi^B, 

w(i;A,B) = b t + ^P^^B) ifi&A, 

j€A 

w(i;A,B) = otherwise. 



Proof. Theorem 3 holds since P is weakly continuous and the discrete topology implies 
that 3(1. a. s. (A)) = 0. Thus (3.15) holds true and the corresponding Bellman equation 
has the form w(i;A,B) = l B (i) + 1^(0 2 Pij w [j>A,B), which is equivalent to (3.16). 

□ 
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Note, that to find a solution for (3.16), one should solve a system of linear equations 
with a non-zero determinant. Moreover, notice that the square submatrix P := (py)yeA 
in (3.16) is contractive since m(A) < oo, so even for large-scale problems efficient 
numerical methods can be applied to solve the problem with any given precision. 

Let us mention what the condition m(A) < oo means in graph-theoretical terms for 
a dtMC. The adjacency graph of a dtMC is a directed graph (V,E), where with V = X 
and the set of edges E is such that (i, j) e E if and only if py > 0. To an arbitrary 
element ieAwe can assign a positive number m h which is the length of the shortest 
path in the graph (V,E) from i toA c . Clearly, it holds that m(A) = supm ; . Moreover, 

ieA 

from this characterization it can be easily seen that m(A) < n(A) if m(A) is finite. As a 
result, if ||u m (a)+iG;A)II = 1 it follows that m(A) = oo and that 

l.a.s.(A) =A KA)+1 = {u KAj+1 (-,A) = 1}. 

3.5. Verification of simplicity of A and determination of l.a.s.(A). Let us summarize 
the methods developed for the solution of the infinite-horizon reach-avoid problem in 
the previous sections. Assume that A, B e 83 are disjoint and let us focus on the case 
when the set A is compact and the kernel P is weakly continuous. If A is simple, it 
follows from Theorem 1 that the solution of (3.3) is unique and that m(A) < oo - the 
solution can be found as in Proposition 2. If A is non-simple, the solution of (3.3) is not 
unique and m(A) = oo, thus Proposition 2 cannot be applied directly. In the latter case, 
there are two approaches to solve an infinite-horizon reach-avoid problem over a non- 
simple set A: if P(x,dy) = p(x,y)/x(dy) is an integral kernel and ju(d(l.a.s.(A))) = 0, 
then Theorem 3 allows formulating an equivalent problem over the simple compact 
set A\ (l.a.s.(A))° . Otherwise, one has to synthesize an appropriate 5-locally excessive 
function to apply Theorem 2. 

All the instances discussed above depend on the fundamental issue of whether a 
given compact set A is simple or not. In general it is hard to provide an analytical 
answer to such a question, and no known general automatic procedure enables com- 
puting absorbing sets exactly. On the other hand, the "if and only if" nature of the 
results in Theorem 1 implies that this issue is not a limitation that is specific to the 
techniques presented in this paper: on the contrary, any other method aiming to solve 
a general infinite-horizon reach-avoid problem is bound to check the simplicity of a 
given set A. 

Let us discuss instances of dtMP for which the l.a.s.fA) of a given set A can be found 
explicitly. The case of dtMC, as discussed in Remark 5, has been recently extended to 
a subclass of dtMP with integral kernels in [TA12, Chapter 4.2]. In both instances all 
the conditions in Theorem 3 are satisfied, thus the reach-avoid problem can be solved. 

Given additional knowledge on the structure of a dtMP, it may be easier to verify 
the dual problem, namely the simplicity of a given set A: if P is ^-irreducible [MT93, 
Chapter 4], then A is simple whenever y(A c ) > 0. If i/> is the maximal irreducibility 
measure, then A is simple if and only if <p(A c ) > 0. However, notice that for a given 
dtMP the verification of its irreducibility can represent an even harder requirement 
than the verification of the simplicity of a specific set A. Moreover, observe that any 
dtMP admitting two disjoint non-empty absorbing sets is not irreducible, which points 
out the conservatism of this condition. 

An additional example where further knowledge on the structure of dtMP may shed 
light on the absorbance of its sets is provided in [AKM11]. As already mentioned, an 
automaton specification j4 over a dtMP = {X , 0S,P) can be verified as a reachabil- 
ity specification over the product j4 x , which is again a dtMP The discrete structure 
of the automaton j4 can be exploited in order to determine absorbing sets within the 
product dtMP j4 x j*f. 
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Furthermore, analytical methods can be applied to find absorbing sets. If the dy- 
namical system representation of a dtMP (2.2) is known one can try to characterize 
its absorbing sets, as the examples of Section 4 will display. Also, for integral kernel 
P(x, dy) = p(x, y )/x(dy) with density p given explicitly, one may try to check for sim- 
plicity using the following result. 

Proposition 3. [TA12, Proposition 3] For x e 3£ define s(x) = jyel: p(x,y) > 0}. 
A set A e Z% is absorbing if and only if ju(s(x) \ A) = 0, for all x e A 

Finally, although in general the verification of the simplicity of a given set is not 
a decidable procedure, the following method can be applied. Let us consider the se- 
quence (A„) n > defined in Lemma 1. If A n = for some n e N, then clearly A is simple. 
Although the definition itself requires a precise characterization of u„(-;A), only the 
computation of P(x, •) is needed in (3.7), instead of consecutive integral iterations 
over value functions. Let us now introduce an approximate approach for the compu- 
tation, using the concepts in Section 2.3: leveraging the procedure in (3.7), we have 
that A = A and that A n+1 = P>j [XA„] . Let us select a precision level 5 e (0, 1) and 
construct a sequence of supersatisfaction sets as follows: 

< +1 =P>i- 5 [XA* n L A* =A. 

By construction, A n c A* for all n > 0, thus A is simple whenever A* = for some neN. 
Notice that the conditions required to implement the procedure are very general. Let 
us discuss its strong and weak points: 

• If the exact form of P is given, then the sets A n can be characterized explicitly. 
The simplicity of A is verified if the sequence (A n )„> eventually contains only 
empty sets. On the other hand, if A is non-simple, then the set l.a.s.(A) can be 
found whenever A n = A, 1+1 ^ for some n e N. Clearly, in such a situation it 
holds that l.a.s.(A) = A n . Finally, if A is non-simple, whenever A n+1 is a strict 
subset of A„, one can compute l.a.s.(A) =A 00 as an intersection of the sets A n . 

• If only an approximate characterization of P is available, the simplicity of set 
A can be verified for sufficiently small 5 and sufficiently large n. However, 
it is not clear how big n should be taken to ensure that A* = for a given 
precision level 5. Due to this reason, it is extremely important to have an a 
priori upper-bound on m(A), provided the latter is finite (cfr. the discussion 
on m(A) for dtMC in Section 3.4). Furthermore, the non-simplicity of set A 
cannot be verified: because of the errors in the computation of A*, the case 
A n = A n+1 7^ cannot be exactly characterized. 

Let us further remark that, for all n > 0, A* provide an overestimation for the set 
l.a.s.(A). If a non-trivial underestimation is available as well, the following result can 
be established. 

Proposition 4. Under Assumption 1, let A be a compact, non-simple set and let sets 
C,D e be such that C C l.a.s.(A) C D, where D is open. Define the sequence u n as 
follows: 

(3m |» n +iW = u W + l A (x) ■ &>[l A \ D (x)u n (xj] , 

\u (x) =l A (x)P(x,D). 

For any n > and x e X it holds that 

ii M m + 2 |_s_| a a m+1 -l 

(3.18) u(-;A) - fi n (.) < p U*J + — • M (D \ C). 

11 11 1 — p 1 — pa— 1 

where m := m(A\D) < oo, p := p(A\D) < 1, and a := sup {p(x,y)|x,y eA}. 
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Proof. The proof is given in [TA12, Section 3.4]. Shortly, the functional sequence in 
(3.17) is designed to approximate to solution of the invariance problem. The first 
term in the right-hand side of (3.18) comes from the upper-bound on the difference 
||u(-;A)— u(-)|| where u := lim £!„ (see [TA12, Theorem 3]) and the second term comes 

n— *oo 

from the upper-bound on the difference \\u — u n \\ (see [TA12, Proposition 2]). □ 

Proposition 4 can be easily extended to be valid over reach-avoid value functions as 
well as dtMP with arbitrary integral kernels, so that Assumption 1 in the statement can 
be relaxed. However, this goal is not pursued in this paper since no procedure to find 
an under-approximating set C is available up to our knowledge. Indeed, any method 
giving a non-empty candidate for C would directly establish the non-simplicity of A. 

We conclude the discussion in this section with the following practical observation: 
in practice stochastic kernels for a dtMP either are extracted from finite data coming 
from measurement experiments, or derived from some underlying analytical model. 
In the latter case, the model gives an additional knowledge on the structure of a dtMP 
which can be further used along the lines discussed in this section to find the largest 
absorbing subset of a given set or to verify the simplicity of such set. Conversely, when 
no underlying model is known and kernels are interpolated exclusively from mea- 
surements data, any kernel resulted via an interpolation technique can be negligibly 
perturbed in order to yield absence of absorbing subsets of given compact sets (see 
discussion after Proposition 2). 

4. Case studies 

4.1. A one-dimensional affine Gaussian system. Let X = K be endowed with the 
standard topology and let 9& be its Borel cr-algebra. Consider a sequence (£ n )„>o of 
iid standard normal random variables and define a dtMP as 

(4.1) X n+1 = (a + nX n ) + (l3 + aX n )-t; n , 

where a, /3, /i, a e K are parameters and X = xel. In order to study the probabilistic 
invariance problem for this affine Gaussian model, let us select a compact set A in K. 
Let us focus on how the structure of the dynamics are affected by the choice of the 
parameters. In order to avoid trivial constant dynamics, let us assume that at least 
one of the parameters a, — l,a is non-zero. If /3 + aX n ^ the distribution of 
X n+1 admits the whole state space K as its support, thus for A to be non-simple it is 
necessary that point /c:=-^eA We then assume that a ^ 0, since clearly if a = 
any compact set is simple. Moreover, for A to be non-simple the state k has to be 
absorbing, so from (4.1) it must hold that k = a + fiK, so a = (1 — /x)zc. Since by 
Theorem 1 the solution of the invariance problem on simple sets is trivial, we focus 
on the case when k is absorbing and select the parameters a := (1 — h)k, p := — <jk, 
where k e K is an arbitrary state. The update equation (4.1) takes the new form: 

X n+1 -k = n{X k -k) + a{X k - K)E, k , 

and by applying a shift on k, without loss of generality we can focus on the following 
model: 

(4.2) X n+1 = fjX n + aX n ■ l n . 

In the latter equation a can be assumed to be positive, since £ n has a symmetric 
distribution. The kernel associated to the dtMP (4.2) is weakly continuous and takes 
the following form: 

I _ ,2 

-^=je-^dt , if* ,4 0, 

A 

1 A (0) ,ifx = 0. 
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Since the compact set A is non-simple if and only if e A, let us consider the invariance 
problem for the set A = [—1, 1]. The discussion above suggests that \.a.s.(A) = {0}, so 
u(0;A) = 1. For x/0, let us relate the original process X to the random walk (see e.g. 
[Dur04, Chapter 4].) Define Y n := logpfj, so the update equation becomes: 

Y n+1 =Y n + log | /i + cr£ n I, 
where Y = y := log|x|. The expected value of the increment of the random walk 

h(/i,cr) := ElogljU + cr^l 
determines its asymptotic behavior. In particular, limsup7 n = +00 a.s. if a) > 0, 

n— >oo 

and lim Y n = —00 a.s., if a) < [Dur04, Chapter 4]. As a result, if the values 

n— *oo 

of the parameters ju, a are such that a) > 0, we obtain that, for any x/0, the 
following holds: 



u(x;A) = P x \ suppq < 1 = P y \ suplog|X n | < = 0, 

( n>0 J ( n>0 J 

which allows to conclude that so that in this case iz(x;A) = l J0 j(x). We are left with 
the case a) < 0. If we represent the kernel in the integral form P(x, dy) = 
p(x,y)/i(dy), we obtain 1 = P(0, {0}) = p(0, 0)ju({0}), so necessarily 

mu({0}) = /x(a(l.a.s.(A))) > 0, 

thus Theorem 3 cannot be applied. We then resort to Theorem 2, which requires 
finding a 5 -locally excessive function. 

Let us fix jU, o and consider g q (x) := |x| q , for q > 0. If we define b(q) = E\[x+<jE !l \ q , 
then clearly 0^g q [x) = b(q) ■ g q (x), so g q is 5-locally excessive if and only if b(q) < 1. 
We obtain b(0) = 1 and b'(0) = fi(ju, <j). Recall that we are now interested in the 
case h(n, a) < 0, which leads to conclude that there always exists a q > such that 
the function g 9 (x) = \x\ q is 5-locally excessive. Hence, for h(fi,a) < and such q, 
Theorem 2 can be applied to find the solution of the invariance problem using g q as a 
1 -locally excessive function. More precisely, according to (3.13) adapted to the special 
case of the invariance problem, we obtain: 

< w{x;A, (- v^, Ve)) - u[x;A) < e, 

and function w(x; A, (— ife, ^fe)~) can be computed, since m(A \ (— Zfe, ^/e)) < 00 as it 
follows from Theorem 2. 

Finally, let us add a comment on the lack existence of a 5-locally excessive function 
for a weakly continuous dtMR Consider the case in (4.2) with parameters cr) > 0, 
so that u(x;A) = l{o}M- If there existed a function g that is 5-locally excessive on 



5 ' 

some neighborhood of {0}: this leads to a contradiction. 



A = [ — 1, 1] then u[x;A) > u(x; {g < 5}) > 1 - which implies that u(x;A) > in 



4.2. A two-dimensional non-linear Gaussian system. Let us provide a more com- 
putational example for the application of the methods developed in this work. Let 
X = K 2 be endowed with the standard topology, and consider a dtMP with dynamics 
given by the following system of non-linear difference equations: 



(4.3) 



X 1<n+1 = 0.5X 2:n (3X 2 n + 2X 2 2 n - 0.5) + 0.6tj„ ^ 2 n +X 2 n , 

X 2 , n+ i = 0.9X, n (2X 2 n + 4X U X 2 ,„ + 3X 2 n - 0.5) + 0.6^ n Jx 2 l n +X 2 2 n , 



where [X lfi ,X 2:0 ) = (x^Xq) = x. Here (Tj n ) n > and [(k\>o are independent sequences 
of iid standard normal random variables. The process is weakly continuous with the 
origin {0} as the absorbing set and its kernel can be expressed explicitly as in Section 
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4.1. We are interested to solve the infinite-horizon invariance problem over the com- 
pact set A = [-0.6,0.6] x [-0.6,0.6]. Since l.a.s.(A) = {0}, the set A is non-simple, 
thus Theorem 3 cannot be applied as discussed in Section 4.2. It is thus necessary to 
find a 5-locally excessive function on A. Let us start by discussing the behavior of the 
process X on the phase plane. For x far from the origin, the non-linear terms (appear- 
ing in brackets in (4.3)) play a more important role then the linear ones, whereas for x 
close to the origin the situation is reversed. We then expect that a function measuring 
the distance from the origin may be locally excessive. For this reason, we consider 
g(x) = ||x|| 2 , which leads to the following form for g?g: 

@'g(x 1 ,x 2 ) = — (144*2 + 197x1 - 474xlxl + 1098x^ - 648x x x 3 ) 
(4.4) 200 

+ (2592x 3 x 3 - 586x* + 5136x?x^ + 3888XT + 1658x^). 

200 12 2 12 2 2 

It holds that {g < 0.25} c g hence g is 5-locally excessive on A, with 5 = 0.25. Figure 

1 shows sets {g < 0.25}, g g , and A Set A intersects g g , which can be interpreted as 

follows: starting from a state x & A(~) g g , process X exhibits contractive dynamics, 

whereas for x eAflf the difference 2? g(x) — g(x) is positive and gets larger as 

||x|| grows, hence trajectories initialized in x &A<lS g expand away from the origin. 

Based on this consideration we expect clear differences between the values of function 

u(x; A) for x e .A n <?„ andxeAn^. 




Figure 1. Infinite-horizon invariance problem over set A. The bound- 
aries of sets g g (dark blue curves), A (brown square) and {g < 0.25} 
(cyan circle). 

We apply the decomposition technique in Theorem 2, where by selecting an e = 
0.02 we obtain that < w(x;A {g < 5 • 10" 3 }) - u(x;A) < 0.02. To simplify the cal- 
culations, we consider the function w{x;A,B) for B = (-0.05.0.05) x (-0.05,0.05) c 
{g < 5 • 10~ 3 } and as a result we have < w(x;A,B) — u(x;A) < 0.02. To compute 
the values of function w we use the bounds provided in Proposition 2: in this case 
m(A \ B~) = 1 and a(A \ B) ^ 0.957, so by considering n = 50 iterations we obtain 
< w{x;A,B) - w„(x;A,B) < 0.105. 

Thus far, the methods developed in this paper (in particular Theorem 2), have al- 
lowed us to reduce the infinite-horizon invariance problem over a non-simple set to a 
finite-horizon reach-avoid problem. Let us now mention how the value function corre- 
sponding to the latter problem can be computed. The calculation of the value function 
w n is performed with a target error 0.1, which is achieved by employing a standard 
uniform discretization algorithm [AKLP10] - thus the resulting overall error equals to 
0.207. Based on the time horizon of the problem, and due to the degenerate nature of 
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the kernel in the neighborhood of the origin, and the fine size selected by the partition- 
ing procedure to achieve the small required precision, the computation took 24 hours 
on Intel Core i5, 2.4 GHz with 4Gb RAM. This computational time can be further re- 
duced by leveraging more involved numerical procedures [SA11], which however are 
outside of the scope of this study. 




%2 Xi 



(a) Local excessivity of g on the set A 




(b) Invariance value function 



Figure 2. Results for the infinite-horizon invariance problem on the 
set A Graphs of functions £?g — g (a) and u (b). 



The first goal of this case study was to show that a infinite-horizon problems can 
be solved efficiently, with strict bounds on the error, even in the case of nonlinear 
dynamics and kernels which admit non-trivial absorbing sets. The use of the decompo- 
sition technique has also allowed us to avoid computations over the neighborhood of 
the absorbing set (0, 0) where the kernel P degenerates. In particular, it is important 
for numerical methods based on the discretization of the state space, since their error 
bounds depends on Lipschitz constants of densities. Moreover, with this approach the 
error of computation can be made as small as needed by varying the error e related 
to the decomposition, the number n of iterations for the reach-avoid problem, and the 
grid size for the discretization. 

As already mentioned, the choice of the set A with regards to the excessive re- 
gion plays an important role. On Figure 2(a) one can observe large positive values of 
^g( x ) — gt x ) for x close to points (—0.6,0.6) or (0.6,0.6). We expect a diverging 
behavior of X when starting in that region. This fact is clearly shown on Figure 2(b) 
where the invariance value function u takes the smallest values exactly in that region. 
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5. Conclusions 

This work has provided a general framework for the study of formal algorithms for 
PCTL verification of discrete-time Markov processes over general state spaces. The 
main focus of the article has been placed on the verification of infinite-horizon PCTL 
specifications, both in terms of characterization of the given PCTL formula and in terms 
of precise numerical computation of the corresponding value function. It has been 
shown that structural properties of the stochastic kernel, namely the possible presence 
of absorbing subsets of given sets, are crucial for problems over the infinite horizon. In 
particular, the solution of the invariance is either trivial (on simple sets) or extremely 
complicated (on non-simple sets). This has lead to criteria to distinguish such instances 
and to techniques to tackle the latter case - these techniques have been illustrated by 
two case studies. 

The outcome of this work is that infinite-horizon problems cannot in general be 
solved exactly or algorithmically. However, precise reduction of these problems to 
finite-horizon analogues allows tapping on techniques for the latter, thus inheriting 
their scalability. This leads to an emphasis on the verification of the simplicity of a 
given set and on the development of procedures to find 5 -locally excessive functions. 

These questions represent compelling goals to the authors and are to be further 
pursued in future work, along with the application of the developed methods to other 
classes of specifications (beyond PCTL). Furthermore, extensions to continuous-time 
and control-dependent models are also deemed research worthy. 
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6. Appendix 

Theorem 1 requires the compactness of the set A and the weak continuity of the 
kernel P, however some of the relations between statements in this theorem are true 
in the general case as it is shown in Figure 6. First of all, for the pair 1 ) <=> 2) it is 




Figure 3. Generalization of the relations between statements of The- 
orem 1. 

clear that 2) is a stronger statement in general. Moreover, from the proof of Theorem 
1 it clearly follows that 1) => 2) without any assumptions on A and P. For 3) <=> 4) the 
following holds: 

Proposition 5. 5 Equation (3.5) admits a unique solution if and only ifu[-;A) = 0. 

Proof. Equation (3.5) is linear, so if its solution is unique it is the trivial zero solution. 
Since u(-;A) is one of solutions, u(-;A) = 0. 

Conversely, let us suppose that u(-;A) = and let / el be any other solution of 
(3.5), so that ||/ 1| > 0. Clearly, the function / := ^ is also a solution of this equation 

and / < 1. As a result, it holds that / < u = (see Remark 1) so that / < 0. On 
the other hand, — / is also a solution of (3.5) due to the linearity of the equation and 
— / < u = which leads to / = 0. However, we have ||_f 1| = 1 by definition, hence we 
come to a contradiction. □ 



This proposition generalizes a result from [RCSL10, Proposition 9], where the trivial invariance was 
shown to be sufficient for the uniqueness over a smaller class of functions. 
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Now we only left to discuss relations between 2), 3) and 5). From contraction 
mapping theorem it follows that 2) => 3). Moreover, if u(-;A) = then A is simple 
since l.a.s.(A) = {u(-;A) = 1} is empty in this case. As a result, all the relations in 
Figure 6 are true. Let us provide examples that other relation does not hold when 
either A is not compact or P is not weakly continuous. We first show that the weak 
continuity is not sufficient. 

1. Let us show that 3) +5) J> 2). Consider an example from Section 4.1 given by 
the equation (4.2) with = and h(0, a) > 0. Let us choose the set A = [— 1, 1] so as 
it has been proved, u(x;A) = 1{ 0} (x). Let us putA = A \ {0}, i.e. it is not compact. By 
induction it can be proved that iz„(x;A) — iz„(x;A) = lj ;( x ) f° r all n > since it holds 
for n = and 



u n +i{x;A) - u n+1 (x; A) = l A (x) 



u n {y;A)P{x, Ay)-l A {x) 



u„(y;A)P(x,dy) 



r 



P(x,dy) = l, 0} (x). 



= l, 0} (x)+l i (x) 

[0} 

Note, however that if / e B is continuous on A = [—1, 1] so is J^/ due to the structure 
of the kernel & . As a result, functions u n (-; A) are continuous on A and since u„(0;A) = 
1 for all n > it holds that ||u n (-;A)|| = ||u n (-;A) - 1 {0 }(-)II = 1- Although the set A is 
simple, its invariance value function u(-;A) = and the uniqueness for the solution of 
(3.5) holds, we have m(A) = oo which proves that 3)+5) 2) in general. 

2. Let us show that 5) a> 3). Following the same lines as above, we consider 
(4.2) with fi = and fr(0, a) < 0. As it has been discussed in Section 4.1 the function 
u(x;A) for A = [—1, 1] is positive in the neighborhood of 0. Still, it holds that iz(x;A) = 
u(x;A) — l{o}M for A = A \ {0} so the invariance value function for the simple non- 
compact set A is non-trivial and hence the solution of (3.5) is not unique. 

Finally, we can show that if A is compact but the weak continuity assumption on 
P is relaxed then 3) +5) =£> 2) and 5) a> 3). To do this, one should make similar 
considerations as in 1. an 2. for the same kernels, just redefining P(0, {2}) = 1. 



